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ABSTRACT 

Compared to the learning that occurs in most engineering courses, the learning that occurs in 
design courses is more dependent on students, and less dependent on instructors. Because typical 
course evaluations are instructor-centric and do not provide information about students’ contri¬ 
butions to their learning, we developed a supplemental evaluation to assess student actions and 
attitudes important to a quality design experience. We detected statistically-significant, logical 
shifts in self-reported practices and attitudes as student cohorts progressed through design proj¬ 
ects. Factor analysis showed that the evaluation questions could be grouped into eight thematic 
categories, with most of the questions assessing student ability to function independently in un¬ 
certain situations, self-perception of maturation and achievement, and acceptance of responsibil¬ 
ity for learning. Accepting responsibility for learning and believing that design experiences helps 
transition from being a student to being a professional were correlated with high student ratings 
of overall instructor performance and overall learning experience. 

Keywords: design, assessment, professional skills, evaluations of teaching. 


INTRODUCTION 


Design courses are, in many respects, different from other engineering courses. While students 
may consider traditional courses as discrete or compartmentalized “units” of learning or concepts, 
design experiences overtly require students to draw from and apply multiple types of accumulated 
experiences and knowledge. Design courses require students to complete a project that is usually 
larger than students have previously completed, with continued student effort over a longer pe¬ 
riod of time than typical class projects, and with more process documentation than students are 
accustomed to providing. Furthermore, design problems are open-ended, and the projects are, in 
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large part, student-driven. In helping students meet these challenges, design instructors guide the 
transition from operating as a student to operating as a professional. 

The factors that make design courses challenging for students also make design courses challeng¬ 
ing for educators. Design instructors must draw from a wide range of experiences and knowledge 
to provide guidance to students in multiple technical areas and in project management. Compared 
to instructors of traditional lecture courses, design instructors often spend more time interacting 
with students one-on-one or in small groups. Design instructors who teach multi-course sequences 
invest their effort in developing the abilities of their students over a longer period of time than 
do instructors whose course enrollments change from term to term. Perhaps the most important 
challenge of instructing a design course is achieving the fine balance between providing specific 
assistance/guidance and allowing students to direct their design process and make (and learn from) 
mistakes as they navigate the open-ended nature of the problem. In order to let the students “learn 
by doing,” design instructors must relinquish some control of the learning process to the students, 
while still maintaining high standards for the quality of the project deliverables. Compared to the 
learning that occurs in traditional engineering courses, the learning that occurs in design courses 
is more dependent on students and their self-directed activities. 

Many academic institutions use formal student evaluations of teaching to obtain feedback on 
perceived course quality and instructor performance. A good deal of research has demonstrated 
that student evaluations of teaching reflect student opinions validly, reliably, and usefully [1, 2, 3], 
Traditional evaluation forms tend to be focused on the course instructor and their practices (e.g., 
“Please rate the overall performance of the instructor,” “The instructor gave well-organized lectures,” 
“The instructor was well-prepared for class,” etc.). These assessment items are most relevant to 
traditional lecture-based courses in which the instructor’s actions and content organization largely 
set the pace and direction of student learning. However, these items do not provide information 
relevant to many of the special challenges faced by design students and instructors. Most notably, 
the instructor-centered nature of these evaluations provides little or no information about student 
contributions to their own learning. 

The assessment of student design performance has been a focus of increased work over the 
last several years [4, 5, 6], including a coordinated effort undertaken by the Transferable Inte¬ 
grated Design Engineering Education Consortium [7]. This consortium has identified multiple 
roles and holistic behaviors of an engineer, including technical, interpersonal, and professional 
areas [7], and identified four broad areas (including, for example, learner development and so¬ 
lution development [8]) within which instructors could assess student performance. Programs 
seeking ABET accreditation must assess the design skills of their students, since some of the 
fundamental program outcomes criteria (/'.e., the “a through k” criteria) required by ABET are 
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specifically focused on design [9], Assessing the learning that occurs during complex tasks 
(involving many components/aspects, with complicated relationships between aspects [10]), 
such as the design process, is challenging. Non-traditional measures such as concept maps [11] 
or causal influence diagrams [10] may help instructors and programs measure how well students 
have mastered design skills. However, summative demonstrations of student skills and technical 
analyses of project deliverables, again, typically yield little or no information about how stu¬ 
dents are contributing to their own learning. Having an instrument that gives some indication 
of how engaged [12] students are in the learning-by-doing design process, and some indication 
of whether a fundamental level of independent metalearning ( i.e learning about learning [13]), 
would be helpful to design instructors. 

In order to assess some of the student actions and attitudes that are important to a successful 
design experience, but that tend not to be addressed through formal student evaluations of teach¬ 
ing or through typical course/program-related assessments of student design performance, we 
developed a supplemental design evaluation. We hoped the supplemental evaluation would provide 
information to design instructors seeking to better understand how students perceived the balance 
between receiving specific, overt project guidance and directing their own design processes and 
learning. We also hoped that correlational analysis of responses to various items on the evaluation 
would provide insight into self-reported attitudes and perceptions about design. This paper describes 
the development and administration of the evaluation, and briefly describes how evaluation results 
were used by design instructors to improve the design experience for students as well as to assess 
the evaluation itself for revision and future use. 


METHODS 


Supplemental Evaluation Instrument 

The first version of our supplemental evaluation consisted of 25 questions regarding student 
practices, student responsibilities, and instructor roles. Because traditional student evaluations of 
teaching already provide information regarding student perceptions of the quality their educational 
experiences, and since we sought to obtain new types of information about the students’ educa¬ 
tional experiences, we avoided rating scales that described levels of quality. Instead the rating scales 
were designed to provide information about the prevalence of activities, amount of overt guidance 
provided, or perceived balance of responsibilities. For example, rather than asking students to rate 
the quality of the instructor's technical advice as excellent, good, fair, or poor, the supplemental 
evaluation asked: 
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“When you discussed design and technical issues with the instructor, they: 

• told you exactly what you should do 

• tried to convince you to take particular actions 

• provided suggestions but did not make decisions for you 

• listened to your concerns but gave no suggestions 

• did not seem to understand your concerns and gave no suggestions.” 

Rating systems such as the scale shown above can be used as instructor goals change over the 
course of a design project or series of design courses. For example, an instructor might intend to 
give students more guidance early in the design experience, and less later on. The supplemental 
evaluation would be able to provide useful feedback to the instructor at both the early and late 
stages of the design experience. Other questions and types of rating systems that were intended 
to provide useful information throughout a design experience were, for example: 

“Taking this course is helping me make the transition from being a student to being a 
professional. 

• Strongly disagree 

• Disagree 

• Neither agree nor disagree 

• Agree 

• Strongly agree.” 

“As this course/series of courses progressed, I had to make choices or decisions even though I 
didn’t know everything there possibly was to know about an issue. 

• Almost never 

• Not very often 

• Often 

• Very often 

• Almost every day.” 

The full supplemental design evaluation instrument used in this study will be downloadable from 
the AEE website as supplementary material for this paper. 

Evaluation Administration 

The supplemental design evaluation was administered in paper form to biomedical engineering 
and civil engineering seniors in their respective capstone design courses at Rose-Hulman Institute of 
Technology, at the conclusions of the Fall and Winter (civil engineering), and Fall, Winter, and Spring 
(biomedical engineering) 2006/2007 terms (IRB approved protocol #RHS0054). Brief descriptions 
of these design courses follow, to provide some understanding of the students' design activities: 
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during the Fall quarter, civil engineering seniors preferenced projects submitted by external clients. 
Teams then developed a written project proposal that summarized the client’s goals and constraints, 
outlined their plan for identifying the problem and designing a solution, and confirmed the scope 
of work. The teams then spent the remainder of the Fall quarter and all of the Winter quarter inves¬ 
tigating the client’s needs, developing and evaluating options, and developing detailed designs for 
the selected options. During the Winter quarter, they were required to deliver an oral presentation 
and submit a written progress report to the client presenting their work thus far and seeking ap¬ 
proval to design the recommended options. During the Fall quarter, teams of biomedical engineer¬ 
ing seniors selected projects with corporate clients or individuals within the community, worked up 
at least three distinct design options, created and used merit and feasibility criteria to select one 
option, and completed a full design proposal for that option. During the Winter quarter the teams 
built and tested their designs, and wrote completed design documentation. The completed projects 
were delivered to clients early in the Spring quarter, and during this quarter the students served 
as mentors for teams of junior biomedical engineering students working through their first major 
design-build-test experience and completing their first design documentation. (Brief descriptions 
of the biomedical engineering design projects completed in 2006/2007 can be obtained separately 
from the AEE website.) 

Student participation in the development of the supplemental design evaluation was anonymous, 
voluntary, and uncompensated. Supplemental evaluations were administered by a faculty member 
while the course instructors were out of the classroom. 100% of the students present in each class 
on the days of administration completed the evaluations as requested, yielding 30, 27, and 29 evalu¬ 
ations completed in the Fall, Winter, and Spring by biomedical engineering students (response rates 
ranging from 80% to 94% of the students registered for these courses) and 30 and 46 evaluations 
completed in the Fall and Winter by civil engineering students (response rates of 60% and 94% of 
the students registered for these courses). Because the biomedical and civil engineering capstone 
design experiences are multi-term, the supplemental evaluation instrument was completed multiple 
times by the same two populations of students. 

Student responses to the evaluation items were numerically coded for quantitative analysis. 
For example, the responses “Almost Never,” “Not Very Often,” “Often,” “Very Often,” and “Almost 
Every Day” were numerically coded as -2, —1, 0,1, and 2. Incorrectly-answered items (e.g., an item 
left blank or an item with two different responses marked) were omitted from the data set. The 
responses were then investigated using correlational and factor analyses (principal components 
analysis with varimax rotation) in SPSS (v.11, SPSS, Inc., Chicago, IL). The numerical data were not 
normally distributed and did not meet the assumptions inherent in the (parametric) Pearson cor¬ 
relation coefficient [14], so the nonparametric Spearman’s rho correlation coefficient was used in 
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this investigation. Similarly, the nonparametric Mann-Whitney test was used to statistically compare 
differences between (numerically-coded) responses from different student groups. 

The full supplemental evaluation instrument, as administered in 2006/2007, can be obtained 
separately from the AEE website. 


RESULTS AND DISCUSSION 


Upon examining the full set of data from all completed evaluations, we discovered that student 
responses to five pairs of evaluation items were moderately correlated (Spearman’s rho > 0.5; Table 1). 
The observed correlations were logical, considering the items involved. For example, one might expect 
that a student who found themselves solving an unexpected problem might need to make choices 
or decisions even though they didn’t know everything there was to know about that problem—those 
two student actions are likely to be correlated. One might also expect that students who believed a 
design course was helping them make a transition from being a student to being a professional would 
be proud of what they achieved in the class—those two student attitudes are likely to be correlated. 

We detected statistically-significant shifts on a few evaluation items from the Fall student re¬ 
sponses to the Winter student responses ( e.g ., Table 2). Table 2 displays evaluation items for which 
the mean student response at the end of the Winter term was significantly different (or most nearly 
so) from the mean student response on that item at the end of the Fall term. The shifts displayed 
in Table 2 are logical given the pedagogical goals and practices of the instructors of the design 


Evaluation Items Correlation Coefficient 

• I had to objectively evaluate the work of my peers 0.77 

• I had to objectively evaluate my own performance 

• Taking this course is helping me make the transition from being a student 0.53 

to being a professional 

• If asked, you could clearly explain why design courses are a 
defining element of an engineering education 

• I had to solve unexpected problems 0.52 

• I had to teach myself things in order to accomplish our tasks 

• I had to solve unexpected problems 0.52 

• I had to make choices or decisions even though I didn’t know everything 
there possibly was to know about an issue. 

• Taking this course is helping me make the transition from being a student 0.50 

to being a professional 

• I’m proud of what I have achieved in this class 


Table 1: Correlated evaluation items. Correlation coefficients are Spearman's rho; n = 133. 
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Student 

Population 

Evaluation Item 

Shift Direction in 

Item Responses 
(from Fall to Winter) 
Toward: 

Significance 
( p value) 

Biomedical 

engineering 

As this course progressed, we received 
less assistance from instructors and took 
on more responsibility for our decisions 
and actions. 

Stronger agreement 

0.01* 


When you asked technicians/professors 
other than the instructor for help, they... 

Showing student more 
of what to do 

0.03* 


Learning from doing a project is a 
different process from learning from 
a book or notes. Who do you think is 
ultimately responsible for what you 
learn in this type of course? 

More student 
responsibility 

0.05* 


In this class, you had to accomplish 
tasks set by the instructor. Who do 
you think is responsible for what you 
achieved in this course? 

More student 
responsibility 

0.06 


In this course I was allowed to make 
mistakes and then learn from the 
consequences of those mistakes. 

Stronger agreement 

0.10 

Civil 

As this course/series of courses 

More strongly 

0.06 

engineering 

progressed, we received less assistance 
and direction from instructors and took 
on more responsibility for our decisions 
and actions. 

agreeing 



When you discussed design and 
technical issues with the instructor, 
they... 

Less instructor 
direction 

0.09 


As this course/series of courses 
progressed, I had to manage competing 
demands for resources and time. 

Occurring more often 

0.09 


Table 2: Quantifiable shifts in student-reported practices or perceptions: Fall to Winter 
quarters. Significances calculated using the Mann-Whitney test; * p < 0.05; n = 57 for all 
biomedical engineering data except for the “When you asked technicians/professors other 
than the instructor for help, they” item, for which n = 45; n = 76 for all civil engineering data 
except for the “When you discussed design and technical issues with the instructor, they...” for 
which n = 75. 


courses; overall, they tend toward students assuming more responsibility for their learning as their 
capstone design sequence progressed. In addition to detecting general overall trends, the supple¬ 
mental evaluation was able to detect differences between the student populations: for example, the 
biomedical engineering but not the civil engineering student population reported having techni¬ 
cians provide more directed assistance over time, and the civil engineering but not the biomedical 
engineering student population reported having to increasingly manage competing demands for 
resources and time. 
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Many of the same supplemental evaluation items showed similar shifts from the Fall to the Spring 
administrations of the instrument (Table 3, Figure 1); responses to some evaluation items did not shift 
appreciably from quarter to quarter (Figure 1). After each administration, histograms of the student 
responses to each supplemental evaluation item were shared with the design course instructors. 
The evaluation results helped confirm for the instructors when course goals were being achieved 
(/'.e., frame A in Figure 1) and when improvements could be made (i.e., the responses in frame B of 
Figure 1 indicating a lack of pride). Because the evaluation was completed anonymously, it was not 
possible to correlate evaluation results with instructor-evaluated team performance or design quality. 
Seeking evidence of such correlations is one aspect of currently-ongoing work in this area. 

Factor analysis showed that the supplemental design evaluation instrument contained questions 
that could be grouped into eight underlying factors or components that accounted for 73% of the 
variance in the data. Examining the individual items grouped within each factor, we were able to 
describe these factors as shown in Table 3. 

The factor analysis revealed that some evaluation items were evidently understood by students 
in different ways than intended. For example, it appears that students may have considered the item 
“As this course/series of courses progressed, I had to separate personal issues from professional 
responsibilities when working in a team” to be asking whether difficult team conflict occurred, 
rather than asking about a normal part of professional teamwork. Many of the factors shown in 
Table 3 are important to how students are likely to respond to the open-ended, non-instructor- 
centric, self-learning nature of typical design problems. For example, a student who is uncomfort¬ 
able dealing with unexpected situations or problems and who wants the instructor to serve as an 


Evaluation Item 

Shift Direction 
(from Fall to Spring) 
Toward: 

Significance 

When you asked technicians/professors other than the 
instructor for help, they... 

Showing student more of 
what to do 

0.005 

In this course, I was allowed to make mistakes and then 
learn from the consequences of those mistakes 

More strongly agreeing 

0.009 

As this course/series of courses progressed, we received 
less assistance and direction from instructors and took 
on more responsibility for our decisions and actions. 

More strongly agreeing 

0.062 

Learning from doing a project is a different process 
from learning from a book or notes. Who do you think 
is ultimately responsible for what you learn in this type 
of course? 

More student responsibility 

0.072 


Table 3: Quantifiable shifts in biomedical engineering student-reported practices or 
perceptions: Fall to Spring quarters. Significances calculated using the Mann-Whitney test; 
n = 59 for all data. 
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In this course, I was allowed to make mistakes and then learn from 
the consequences of those mistakes. 



Strongly Disagree Neither Agree Agree Strongly Agree 

Disagree Nor Disagree 


Response 

I'm proud of what I've achieved in this class. 



Strongly Disagree Neither Agree Agree Strongly Agree 

Disagree Nor Disagree 


Response 


Figure 1: Histograms of longitudinal biomedical engineering student responses regarding 
learning from mistakes and taking pride in work. Data are reported as percent of respondents 
to each administration of the evaluation. 


overtly directive authority figure is unlikely to enjoy the learning-by-doing aspect of design, and 
may therefore give design courses and instructors poor evaluations. To explore this possibility, in 
the Spring of 2006/2007 we added two questions to the end of the supplemental design evalua¬ 
tion, phrased exactly as are two key questions from the official institutional student evaluations of 
teaching: “Please rate the quality of your learning in this course,” and “Please rate the professor’s 
overall performance in this class,” (response choices for both questions were: Poor, Fair, Satisfactory, 
Very Good, or Excellent). As is standard practice with the official student evaluations, these answers 
were coded numerically to correspond to the integers 1 (“Poor”) through 5 ("Excellent”). 

Some items on the supplemental evaluation were mildly to moderately correlated with student 
ratings of overall instructor performance and overall learning (Table 4). Table 4 provides evidence 
that students who believe they are learning from their design course and who believe the design 
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Factor 

Description 

Percent of Variance in Data 
Attributed to This Factor 

1 

Students’ ability to independently handle situations 
as they arise 

21.4 

2 

Students’ self-perception of maturation and 
achievement 

11.6 

3 

Students’ dependence on the instructor as a guide 

10.4 

4 

Students’ dependence on assistance from others 

7.7 

5 

Students’ acceptance of responsibility for learning 

6.7 

6 

Students’ ability to handle uncertainty 

5.6 

7 

Students’ rejection of responsibility for learning 

5.5 

8 

Students’ motivation by the project or client 

4.5 


Table 3: Main underlying factors or components assessed in the supplemental design 
evaluation. The percent of variance attributed to each factor can be interpreted as a relative 
ranking of the importance of the factor—so, the underlying construct of students’ self¬ 
perception of maturation and achievement explains more of the variation in the data than the 
construct of students' dependence on the instructor as a guide. 


Correlated with rating of 
professor’s overall performance 

Correlation 

Correlated with rating of quality 
of learning in the course: 

Correlation 

Rating of quality of learning in 

course 

0.50 

Rating of professor’s overall 
performance 

0.50 

Strength of agreement with 
“Taking this course is helping me 
make the transition from being a 
student to being a professional” 

0.43 

Strength of agreement with 
“Taking this course is helping me 
make the transition from being a 
student to being a professional” 

0.50 

Strength of agreement with “If 
asked, you could clearly explain 
why design courses are a defining 
element of an engineering 
education” 

0.39 

Strength of agreement with “I’m 
proud of what I’ve achieved in this 
class” 

0.46 

Strength of agreement with “I’m 
proud of what I’ve achieved in this 
class” 

0.37 

Strength of agreement with “As 
this course/series of courses 
progressed, we received less 
assistance and direction from 
instructors and took on more 
responsibility for our decisions and 
actions” 

0.44 


Table 4: Supplemental evaluation items correlated with overall ratings of instructor 
performance and learning. Correlation coefficients are Spearman’s rho; all correlation 
coefficients shown are statistically significant at the p < 0.01 level; n = 29. 


to 
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experience is helping them transition from being a student to being a professional may be likely to 
rate their design courses and instructors better than students who do not believe they are learning 
and growing from the experience. 

Student ratings of the professor's overall performance reported on the supplemental evaluation 
were generally similar to the student ratings of the professor’s performance reported on the official 
institutional evaluation forms; the official ratings tended to be slightly higher (usually 0.1 or 0.2 points 
higher on the scale of 1 to 5). The official student ratings of their overall learning were either similar to 
or lower than the quality of learning reported on the supplemental evaluation. In one course section, the 
learning rating from the supplemental evaluation form was one full point higher (on the scale of 1 to 5) 
than the rating on the official evaluation. This was likely due to different response rates, and sampling 
error associated with small response rates: in this case, the response rate on the supplemental evaluation 
was 94% and the response rate on the (electronically-administered) official evaluation was 32%. 

Information from the supplemental evaluation helped put student comments on the official 
teaching evaluations into a new context. For example, the following comments were submitted in 
response to ‘Please rate the quality of your learning’ in the official teaching evaluations: 

“This class didn’t really ‘teach’ much information, more that it tried to help us use info we 

already had in a new way.” 

“They have formatted the class so that the group is responsible for their own project design 

and decisions to make it work.” 

While these comments were associated with lower numerical ratings (and therefore interpreted as being 
‘negative’ from the student perspective), these comments are encouraging to a design instructor. Design is 
the capstone experience for engineering students and does bring together previous material to undertake 
a larger project, and we do want students to recognize that they are responsible for their projects. 

Information from the supplemental evaluation was used by course instructors as part of their 
continual improvement efforts. For example, in response to learning that some students appeared 
to not be taking ownership of their projects (and were also rejecting responsibility for their learn¬ 
ing), changes were made in the next iteration of the course to explicitly discuss design as a defin¬ 
ing feature of engineering, and to have students model aspects of professional behavior from the 
beginning (the Junior design course) instead of introducing this in the Senior Fall course. One of the 
most important of these aspects was having students sign all submitted documentation, and having 
related discussions of what their signatures mean about the their relationship to the submitted work 
(since their signature means they approve—as an engineer—of the entire contents). Information 
regarding student perceptions of their team’s workload and handling of uncertainty indicated that 
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many students saw their team as having to do more work and handle greater uncertainty than other 
teams. By relocating designated design project time to a large, common room where all teams work 
in the same area, students can see that other teams are facing similar challenges (if on very differ¬ 
ent projects), and this has increased their ability to work with uncertainty because they realize that 
it’s part of the process. An unexpected, but correlated outcome for working in a common room in 
their teams is that students in the current design sequence recognized early on that classmates are 
resources (for information, ideas, etc.) and not competitors in design. Additionally, after learning 
from the supplemental evaluation results that some students did not feel proud of their work, the 
instructors made a concerted effort to give positive feedback and praise when the opportunity arose. 
Specifically, the instructors made an effort to praise students for their hard work and persistence, 
since literature reports indicate praise of this nature may encourage students to focus on learning 
goals (e.g., "I want to figure this out.”) rather than performance goals (e.g., "I want to get an A in 
this class.”) [15]. Students who operate from a learning-based orientation may respond better to 
academic challenge [16], such as the challenges inherent in design projects. 


CONCLUSIONS 


We have developed a supplemental design evaluation that can provide quantitative information 
about whether design instructors' goals are being met—for example, whether students are assum¬ 
ing more responsibility for their learning as the design experience progresses. The supplemental 
evaluation provides information that is not assessed by traditional student evaluations of teaching, 
or through grading/assessment of the technical quality of student-designed deliverables. The evalu¬ 
ation can be used by design instructors to better understand students’ perceived independence, 
responsibility for learning, and prevalence of professional activities. Design instructors can then 
change aspects of their courses (more clearly explaining particular requirements, having different 
discussions about professional responsibilities, asking students to keep track of/reflect on various 
professional work practices, etc.) to encourage desired outcomes. 

We are currently working on revising the initial version of the supplemental design evaluation by 
revising items that appear to be interpreted in multiple ways by students, and eliminating items that 
appear to obtain redundant information. We plan to add items related to an underlying factor of 
student self-assessment of the technical quality of their design product, and to add items to more 
clearly understand student orientations to challenge and ability to overcome academic difficulties 
[15, 16]). Our ultimate goal is to develop and disseminate a short, robust instrument that could be 
used to obtain information particularly relevant to design courses. 
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